Virtual reality (VR) systems are used both within and outside of the video game industry. Displays for VR systems, such as those embedded in a VR headset, typically operate at a minimum refresh rate that is suitable for VR applications. For instance, 90 Hertz (Hz) is a common refresh rate for VR displays. In a “live rendering” scenario, a graphics-based application, such as a video game, outputs frames for rendering at a frame rate that matches the refresh rate of the display, meaning that a new frame received from the application (referred to herein as an “actual frame”) is displayed at every screen refresh. Such a live rendering scenario is often referred to as the application “making frame rate” or “hitting frame rate.”
In practice, an application does not always make frame rate for various reasons. For example, the application may intermittently drop a frame, and/or the application may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). In situations where the application is not making frame rate, a technique called “rotation-only re-projection” can be used to replace missing frames with re-projected frames in a way that accounts for the user's head rotation, making it appear to the user as if the application is making frame rate. Without re-projection, for example, a deficient frame rate from the application may cause in-game stuttering or hitching. In VR applications, where the user is fully immersed in the virtual environment, the user can become nauseous if the application fails to make frame rate and there is no re-projection to compensate for the missing frames. Thus, re-projection is a technique that allows for a better user experience when an application is not making frame rate. Consider an example where the application is outputting frames at half the ideal frame rate (e.g., 45 frames per second where 90 frames per second is the ideal frame rate). In this example, every other frame can be re-projected using pixel data from the most recently-rendered actual frame to create a re-projected frame that transforms the scene (e.g., through rotation and re-projection calculations) to match the re-projected scene to the user's current head orientation. This makes it look to the user as if the scene is moving in a way that is expected given the user's head rotation.
Although rotation-only re-projection mitigates in-game stuttering or hitching, it produces its own unwanted visual artifacts during head rotation, at least in VR systems that use low-persistence displays (e.g., where the display is illuminated for a small fraction of the frame time). For example, although rotation-only re-projection accounts for head rotation, it does not account for objects that move or animate in the scene between frames. This can cause an unwanted visual artifact called “judder” to occur with respect to moving or animating objects. Judder causes the user to perceive a “double ghosting effect” where a moving object (e.g., a bullet or a ball moving across the screen) appears to bounce between two locations (or separate from itself) frame-to-frame. Accordingly, when the user rotates his/her head while re-projection is being used, any moving or animating objects in the scene will judder. Because the rendering performance of the application is unpredictable, the frame rate of the application tends to move in and out of phase with the vertical synchronization (VSync) signals of the display by varying amounts and at random times, and the aforementioned judder of moving objects can become erratic and intolerable to the viewing user.
Provided herein are technical solutions to improve and enhance these and other systems.